Notification device interaction

ABSTRACT

Devices and methods are disclosed for establishing device interaction among electronic devices of a device environment and a notification device interfaced with the user. Notification devices allow the user to communicate directly with the device environment, such as by entering information in a mobile phone that has a wireless connection to the environement. A device of the environment has a first transmitter and receiver for communicating with other devices, such as a notification device within the environment. In the instance where a notification device is located remotely, the device of the environment may also include a second transmitter and receiver for communicating with the remotely located notification device through a remote communications transport. A memory stores interaction rules and translation rules, and a processor implements the interaction rules and translation rules to establish interaction with the notification device. Methods may involve detecting a change of state at a device and then communicating with the notification device in accordance with the interaction rules and translation rules thereafter. Methods may also involve directing a message from a notification device to a device of the environment, such as through the remote communication transport, and then communicating from the device of the environment to additional devices.

TECHNICAL FIELD

The present invention relates to interaction of a device with a user.More specifically, the present invention relates to communicationbetween devices located within a device environment and a notificationdevice interfaced with the user.

BACKGROUND

Electronic devices such as household appliances, audio-video equipment,computers, and telephones operate within a given environment such as thehome of a user. However, these devices function independently of oneanother. The user must initiate actions on the devices to cause thedevices to change to a particular state of operation to thereby performa function desired by the user.

Often, the state of one or more of the electronic devices is related tothe state of one or more other electronic devices within the sameenvironment. For example, a user may be watching television (TV) whenthe telephone rings. The user wishes to answer the call, but toeffectively communicate with the caller, the user must mute thetelevision so that sound from the TV does not interfere with thetelephone conversation. Every time a telephone call is to be answeredwhile the user watches TV, the user must again repeat the mutingprocess. For each call, once the user hangs up the phone, the TV must bemanually unmuted so that the user can once again listen to the TVprogram being watched.

The TV—telephone scenario discussed above is only one example. There isan undeterminable number of these scenarios and devices involved withina given environment. In each scenario the devices do not communicatewith one another and do not coordinate activities, and as a result theuser is overly burdened. The number of electronic devices for ahousehold is continually increasing, and the resulting burden on theuser to manually coordinate states of the devices for given scenarios isincreasing as well.

To address this problem, devices can be configured with communicationabilities so that they can communicate with one another when one or moredevices experience a user driven state change. The communication allowsdevices to coordinate activities within the device environment to lessenthe burden on the user. In certain scenarios, the user needs to be incontact with the device environment. For example, the user may want toknow what activity is occurring among devices of the environment, suchas learning that the curling iron was inadvertently left on when theuser left home.

Therefore, there is a need for interaction of electronic devices withinan environment with notification devices interfaced with the user toallow the user to remain in contact with the device environment.

SUMMARY

Embodiments of the present invention provide communications capabilitiesto the device environment to allow communication between notificationdevices interfaced with the user and the other devices of theenvironment. Notification devices allow the user to directly communicatewith the device environment, such as by dialing numbers or speaking on atelephone. For example, communication may occur between a device of theenvironment and the user's mobile phone which is an illustrativenotification device. The devices of the environment may alert the userthrough the notification device that a particular state of one or moredevices within the device environment are noteworthy. Likewise, the usermay communicate back to the device environment through the notificationdevice to specify that a particular device should acquire a particularstate.

A device within the environment utilizes a first transmitter andreceiver that enable communication with other devices such as thenotification devices through a particular transport. Alternatively, asecond transmitter and receiver may be included in a device to enablecommunication with a notification device interfaced with the user. Thenotification device may be included in the device environment byproximity or by explicit definition by the user, and wireless transportssuch as infrared or radio frequency as well as wired connections andmany other transports are available for use by the notification devicesof a particular environment. A communication transport can be of similarforms such as a telephone line, a wireless transport such as cellular,or a network connection such as Ethernet.

The devices of the environment have memory for maintaining interactionrules that the devices obey to communicate with the notification devicesof the user. Translation rules for translating messages between thedevice environment and the notification devices may be stored in thememory. A processor is included to employ the logic necessary to executethe interaction rules and, if necessary, the translation rules toestablish communication with devices of the environment and thenotification device of the user.

The logical operations of the processor that involve implementing theinteraction rules and translation rules are embodied in methods. Themethods specify how a particular device or group of devices communicateto a notification device. One embodiment of a method involves detectinga change of state at a first device and in response to detecting thechange of state, referencing rules of device interaction that governcommunication between the first device and a notification device thatmay or may not be located remotely from the local proximity of thedevice environment. The rules of device interaction provide anassociation between the detected change of state of the first device,the notification device that is communicated with in response, and thecontent of the communication to the notification device. Rules oftranslation may be provided to indicate instructions for conveyinginformation to the notification device if it is not included in thedevice environment by proximity. In such a case, communication with theremote notification device occurs according to the rules of deviceinteraction and the rules of translation in response to detecting thechange of state.

Another embodiment of a method involves directing a message from anotification device to a first device of the environment, and themessage specifies an instruction for a second device of the environment.The first device may be configured to translate messages between remotecommunication transports of a remote notification device and transportsof the devices of the environment. The first device receives the messagefrom the remote notification device over the remote communicationstransport and translates the received message at the first device toadapt the message for transmission over the transport of the seconddevice. The first device communicates the instruction of the message tothe second device over the transport of the second device. The seconddevice then implements the instruction to cause a change of state at thesecond device as desired by the user interface with the notificationdevice.

The various aspects of the present invention may be more clearlyunderstood and appreciated from a review of the following detaileddescription of the disclosed embodiments and by reference to thedrawings and claims.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a device environment.

FIG. 2 is a diagram showing the major components of an embodiment of aninteractive device.

FIG. 3 is an exemplary operational flow of an interactive devicecommunicating with other devices of the environment.

FIG. 4 is an exemplary operational flow of communication betweeninteractive devices involving a message broadcast to all devices of theenvironment.

FIG. 5 is an exemplary operational flow of communication betweeninteractive devices involving a message directed to a specific device ofthe environment.

FIG. 6 is an exemplary operational flow of communication betweeninteractive devices involving a request and a response to the request.

FIG. 7 is an exemplary operational flow of rule acquisition of aninteractive device.

FIG. 8 is an exemplary operational flow of rule acquisition of aninteractive device involving learning by a device receiving a statechange after a state change of another device.

FIG. 9 is an exemplary operational flow of rule acquisition of aninteractive device involving learning by a device receiving a statechange before a state change of another device.

FIG. 10 is an exemplary operational flow of rule acquisition of aninteractive device involving a request for rules to a device and asubsequent response.

FIG. 11 is an exemplary operational flow of content control amonginteractive devices.

FIG. 12 is an exemplary operational flow of media rights sharing amonginteractive devices.

FIG. 13 is a diagram of a device environment that illustrates thecomplexity that occurs in relation to interactivity among an increasingnumber of devices.

FIG. 14 is a diagram of a device environment including an embodiment ofan aggregator that also illustrates the major components of theaggregator.

FIG. 15 is a diagram of an embodiment of an aggregator illustrating thecomponents for translating among multiple communication transports.

FIG. 16 is an exemplary operational flow of device interaction involvingan aggregator.

FIG. 17 is an exemplary operational flow of device interaction involvingan aggregator that learns interaction rules and translates amongmultiple communication transports.

FIG. 18 is a diagram of a device environment interacting withnotification devices interfaced with a user.

FIG. 19 is an exemplary operational flow of interaction from a deviceenvironment to a remote notification device through a remotecommunication transport.

FIG. 20 is an exemplary operational flow of interaction from a remotenotification device to a device environment through a remotecommunication transport.

FIG. 21 is a diagram of an embodiment of a device for providing adisplay of information about a device environment.

FIG. 22 is an exemplary screenshot of the device of FIG. 21 thatillustrates a device menu and a learn mode menu.

FIG. 23 is an exemplary screenshot of the device of FIG. 21 thatillustrates a learn mode allowing the user to select functionrepresentations on the screen to associate functions of devices.

FIG. 24 is an exemplary screenshot of the device of FIG. 21 thatillustrates a learn mode allowing the user to select functions ondevices that are to be associated.

FIG. 25 is an exemplary screenshot of the device of FIG. 21 thatillustrates a rule display mode for visually conveying the stored rulesto a user.

FIG. 26 is an exemplary operational flow of a learn mode where the userselects functions on the devices that are to be associated.

FIG. 27 is an exemplary operational flow of a device information displaymode.

FIG. 28 is an exemplary operational flow of a learn mode where the userselects function representations on a display screen to associatefunctions of devices.

DETAILED DESCRIPTION

Interaction among devices of an environment permit the devices toperform automatic changes of state without requiring the user toindividually control each device. Through recognition of patterns ofuser behavior, interactive devices can associate the various user drivenevents from one device to the next to effectively create interactionrules. Application of these interaction rules allow the devices toimplement the state changes automatically through communication ofevents between the devices.

A device environment is shown in FIG. 1 and is representative of a smallarea such as within a single household. However, a device environmentmay expand beyond a single area through networking of devices amongvarious areas. This simplified device environment 100 shows threedevices for exemplary purposes, but any number of devices may be presentwithin a given environment 100. The devices of the environment 100 aredevices that customarily appear within the particular type ofenvironment. For example, in a household the devices would include butnot be limited to typical household devices such as a television, VCR,DVD, stereo, toaster, microwave oven, stove, oven, washing machine,dryer, and telephone. These devices are adapted to become interactive asis discussed below.

Each device communicates with the other devices of the environment 100in this example. A first device 102 communicates with a second device104 through a bi-directional communication path 108. The first device102 communicates with a third device 106 through a bi-directionalcommunication path 110, and the second device 104 communicates with thethird device 106 through a bi-directional communication path 112. Thecommunication paths may be wired, wireless, or optical connections andmay utilize any of the well-known physical transmission methods forcommunicating among devices in a relatively small relationship to oneanother.

The communication method used between two devices makes up acommunication transport. For example, two devices may utilize theBluetooth transport, standard infrared transport where line of sight ismaintained, a UHF or VHF transport, and/or many others. Networked areasforming an environment can utilize LAN technology such as Ethernet, WANtechnology such as frame relay, and the Internet. Multiple transportsmay be present in any single environment. As discussed below withreference to FIGS. 15 and 17, a particular device such as an aggregatormay be equipped to translate messages from one communication transportto another. Aggregators are discussed generally and in more detailbelow.

The details of the devices 102, 104, and 106 are shown in more detail inFIG. 2. An interactive device 200 includes a processor 206 thatcommunicates with various resources through a data bus 214. Theprocessor 206 may execute software stored in a memory 208 or may utilizehardwired digital logic to perform logical operations discussed below tobring about the device interaction. The processor 206 communicates withthe memory 208 to apply interaction rules that govern thecommunications. Interaction rules specify when a particularcommunication should occur, the recipients of the communication, and theinformation to be conveyed through the communication. Memory 208 mayinclude electronic storage such as RAM and ROM, and/or magnetic oroptical storage as well.

The processor 206 communicates with a transmitter 212 and a receiver 210to physically communicate with the other devices of the environment. Thetransmitter and receiver pairs discussed herein for the variousembodiments may be separate or incorporated as a transceiver. When aninteraction rule specifies that a communication from device 200 shouldoccur, the processor 206 controls the transmitter 212 to cause it tosend a message. The message may take various forms discussed belowdepending upon the intended recipients. The receiver 210 receivesmessages directed to the device 200. The communications among devicesmay be configured so that each device to receive a message hasidentification data included in the message so that the processor 206determines whether a message is relevant to the device 200 based onwhether particular identification data is present.

Alternatively, other schemes may be used to communicate wherein aphysical parameter of the receiver 210 controls whether a device 200receives the message as one intended for it to be received. Examples ofsuch physical parameters include the particular frequency at which asignal is transmitted, a particular time slot during which the messageis transmitted, or the particular type of communication transport beingused. The transmitter and receiver may be of various forms such as amodem, an Ethernet network card, a wireless transmitter and receiver,and/or any combination of the various forms.

The processor 206 also interacts with the intended functionality of thedevice 200. The device 200 includes components 202 that provide theunique function of the device 200. If the device 200 is a television214, then the components 202 include the circuitry necessary to providethe television function. One skilled in the art will recognize that theprocessor 206 can be separate and distinct from the processingcapabilities of the components 202 or alternatively, may be wholly orin-part incorporated into the processing capabilities of the components202. The components 202 of many devices have digital logic such as anon-board processor of a television 214, CD player 216, stereo system218, dryer 220, or telephone 222.

The processor 206 can control the operations of the components to causestate changes of the device 200. For example, the processor 206 cancause the channel to change on the television or cause the oven topreheat to a particular temperature. Thus, the processor 206 canreference interaction rules stored in memory 208 in relation tocommunications received through receiver 210 to determine whether astate change is necessary or can receive state change instructionsthrough receiver 210 and implement the requested state change.

Additionally, the device 200 includes a sensor 204 for providing statechange information to the processor 206 about the device 200. The sensor204 may be either a physical sensor such as a transducer for detectingmotion or a thermocouple for detecting temperature, or the sensor 204may be a logical sensor. The logical sensor may be a programmed functionof processor 206 or the processor of components 202 or may be hardwiredlogic. A logical sensor may be separate and distinct from the processor206 and/or the digital logic of components 202 and communicate throughthe bus 214, or it may be incorporated wholly or in part in either theprocessor 206 of the processor of the components 202. The logical sensor204 acts as an interface to the digital logic of the components 202 fordetecting the logical state of a component, such as a particular inputthat is active on a stereo, or a particular channel being displayed on atelevision.

The processor 206 receives input from the sensor 204 to determine acurrent state of the components 202 and thereby determine when a changeof state of the device 200 occurs. As discussed below, changes of stateare used to learn interaction rules and implement the rules once theyhave been learned. Implementing interaction rules involves controllingchanges of state at device 200 and/or transmitting change of stateinformation about device 200 to other devices or transmitting change ofstate instructions to other devices.

FIG. 3 shows the basic operational flow of the processor 206 forimplementing device interaction to send a communication from device 200.A state change is detected at the device 200 as described above atdetect operation 302 by the sensor 204. The state change may be a userdriven event, such as a user turning the power on for the television, oran automatically occurring event such as an oven reaching a preheattemperature.

After detecting the change of state, the processor 206 references therules of device interaction stored in the memory 208 to determinewhether a communication is necessary, who should receive thecommunication, and the particular information to include at ruleoperation 304. The processor 206 performs a look-up of the state changethat has been detected to find the interaction rule that is appropriate.The processor 206 then communicates according to the appropriateinteraction rule by sending a message through transmitter 212 atcommunicate operation 306.

FIG. 4 shows an operational flow of a specific type of communicationwhere a device 200 publishes its state change to all devices via abroadcast so that all devices receive the message. A broadcast to alldevices is useful when devices are attempting to learn interaction rulesby observing state change events occurring within the device environment100 during a small interval of time.

The operational flow begins at detect operation 402 where the processor206 realizes that sensor 204 has detected a change of state at device200. The processor 206 then determines that a broadcast is appropriateat determine operation 404. The processor 206 may make thisdetermination by referencing the rules of interaction to determinewhether a broadcast is indicated. If learn modes are provided for thedevices, as discussed below, then the processor 206 may recognize thatit is operating within a learn mode where broadcasts of state change arerequired.

Once it is determined that a broadcast to all devices is appropriate,the processor 206 causes the broadcast to occur by triggering thetransmitter 212 to send the message to all devices of the environment atbroadcast operation 406. As discussed above, messages may be addressedto specific devices by manipulation of a transmission frequency, a timeslot of the transmission, or by including recipient identification datain the transmission. The message contains an identification of thedevice 200 and the particular state change that has been detected.

The devices of the environment receive the message at receive operation408. In this exemplary embodiment shown, the devices make adetermination as to whether a reply is necessary at determine operation410. Such a determination may be made by the devices by referencingtheir own interaction rules or determining that a learn mode is beingimplemented and a reply is necessary because they have also detectedtheir own state change recently. When a reply is necessary, the one ormore devices of the environment send a reply message addressed to thedevice 200 at send operation 412, and the device 200 receives themessage through receiver 210 at receive operation 414.

FIG. 5 shows an operational flow where a message is directed to aspecific device of the environment from the device 200. The processor206 recognizes that the sensor 204 has detected a change of state of thedevice 200 at detect operation 502. The processor 206 then determinesfrom the interaction rules that a second device is associated with thestate change at determine operation 504. The second device may be asubscriber, which is a device that has noticed through a learningoperation that it is related to the first device 200 through aparticular state change event and that the first device should provideit an indication when the particular state change event occurs. Once ithas been determined who should receive a message, the processor 206triggers the transmitter 212 to direct a message to the second device atsend operation 506, and the message includes a notification of the statechange of the first device 200.

The processor 206 may employ additional logic when directing the messageto the second device. The processor 206 may detect from the interactionrules in memory 208 whether the second device should change state inresponse to the detected change of state of the first device 200 atquery operation 508. If so, then the processor 206 includes aninstruction in the message to the second device at message operation 510that specifies the change of state that should be automaticallyperformed by the second device.

FIG. 6 shows an operational flow where a request is made and a responseis thereafter provided. At detect operation 602, the processor 206recognizes that the sensor 204 has detected a change of state of thedevice 200. The processor 206 then determines that a second device isassociated with the state of change at determine operation 604. In thiscase, the processor 206 recognizes that a request to the second deviceis necessary, such as by reference to the interaction rules or due tosome other reason such as a particular learn mode being implemented.

The processor 206 triggers the transmitter 212 to direct a requestmessage to the second device at send operation 606. The request messagecan specify that the second device is to respond by transmittingparticular data that the second device currently possesses in memory tothe device 200. The second device receives the request message atreceive operation 608. The second device prepares a response byobtaining the required information from its memory, sensor, orcomponents. Such information includes interaction rules, its currentstate, its current capabilities, or those who have subscribed to it forstate change events. Once the information is obtained, the second devicesends the response including the information to the first device 200 atsend operation 612.

FIG. 7 is an operational flow of a learning process of the device 200.The device 200 may learn interaction rules that it obeys by observingactivity in the environment in relation to its own state changes.Because the device may automatically learn interaction rules rather thanrequiring that they be manually programmed, a burden on the user islessened. The operational flow begins by observing the environment todetect a state change message at observation operation 702. The statechange message may originate from another device of the environment andis received through the transmitter 210. State changes of the device 200that is learning the rule are also detected through its sensor 204.

After detecting state change messages, the processor 206 learns the ruleat learn operation 704 by associating together state changes that haveoccurred over a small interval of time. For example, a user may turn onone device and then shortly thereafter manipulate another device, andthese two state changes are observed and associated as a rule.Particular methods of learning are discussed in more detail below withreference to FIGS. 8 and 9. The processor 206 stores the rule in thememory 208 where it can be referenced for subsequent determinations ofwhether state changes should occur automatically. The rules are appliedfrom the memory 208 at application operation 706.

FIG. 8 shows the logical operations where the device whose state changeslater in time learns the interaction rule. The operations begin atdetect operation 802 where a first device detects a change of statethrough its state sensor. The first device determines that a broadcastis appropriate and sends the broadcast of the state change to alldevices of the environment at broadcast operation 804. All devicesreceive the broadcast at receive operation 806.

After receiving the broadcast, each device of the environment monitorsfor its own state change. A second device that received the broadcastdetects its state change at detect operation 808 within a predeterminedperiod of time from when the broadcast was received. The second devicethen creates the interaction rule by associating the state change of thefirst device with the state change of the second device at ruleoperation 810. The rule is stored in the memory of the second device sothat the processor can apply the rule thereafter.

At application operation 812, the second device receives the statechange message from the first device and then applies the interactionrule that has been learned to automatically change its stateaccordingly. The second device applies the interaction rule by lookingup the state change of the first device in its memory to see if there isan association with any state changes of the second device. Thepreviously learned rule specifies the state change of the second device,and the second device automatically changes state without requiring theuser to manually request the change.

As an example of this method of learning, the user may turn on the VCRwhich sends a broadcast of the state change. The user shortly thereaftertunes the TV to channel 3 to watch the VCR signal. The TV has receivedthe broadcast from the VCR prior to the user tuning to channel 3, andtherefore, the TV associates the tuning to channel 3 with the VCR beingpowered on to learn the interaction rule. Thereafter, when the userturns on the VCR, the TV automatically tunes to channel 3.

FIG. 9 shows an alternative method of learning where the first device tohave a change of state learns the interaction rule. The logicaloperations begin at detect operation 902 where a first device detectsits own change of state. In response to the change of state, the firstdevice then begins monitoring for incoming state change messages atmonitor operation 904. Subsequently, a second device receives a changeof state at detect operation 906 and broadcasts the change of statemessage to all devices at broadcast operation 908. The broadcast iseffectively a request that any device previously experiencing a statechange add the second device to its subscriber list.

While monitoring, the first device receives the change of state messagefrom the second device at receive operation 910. Because this messagewas received within a predetermined amount of time from when the firstdevice detected its own change of state, the first device creates aninteraction rule at rule operation 912. The first device creates theinteraction rule by adding the second device and state change to itssubscriber list that is associated with its state change. Subsequently,the first device detects its state change at detect operation 914 andthen directs a message to the second device at message operation 916 inaccordance with the interaction rule learned by the first device.

The message to the second device provides notification that the seconddevice should perform a particular state change. Once the message isreceived at the second device, the message is interpreted, and thesecond device automatically performs the appropriate state change withno input from the user at state operation 918. As an example of thismethod of learning, the user turns on the VCR which begins to monitorfor a state change broadcast. The user tunes the television to channel 3shortly thereafter, and the television broadcasts the state change. TheVCR receives the broadcast and associates the TV to channel 3 statechange with its power on state change. After the rule is created andwhen the user powers on the VCR, the VCR executes the rule by sending amessage with instruction to the TV. The TV implements the instruction toautomatically tune to channel 3.

FIG. 10 shows another alternative learning method. For this method, itis assumed that a device already has one or more interaction rules. Thelogical operations begin at send operation 1002 where a first devicesends a request to a second device. The request is for the interactionrules stored by the second device. The rules of the second device may berelevant to the first device for various reasons such as because thefirst device is involved in the interaction rules of the second deviceor because the first device is acting as an aggregator that controlsinteraction of the environment. The details of the aggregator arediscussed in more detail below.

After the first device has sent the request, the second device receivesthe request at receive operation 1004. The second device then retrievesits interaction rules from memory at rule operation 1006. The seconddevice then sends a reply message to the first device at send operation1008 that includes the interaction rules of the second device. The firstdevice receives the reply message with the rules at receive operation1010 and stores the rules in memory at rule operation 1012. Thereafter,the first device can apply the rules stored in memory to control statechanges upon user driven events at application operation 1014.

Device interaction also permits additional functionality among thedevices of an environment such as the control of media content to beplayed within the environment and the control of device settingsdependent upon the particular media being played. FIG. 11 shows thelogical operations of device interaction involving a device, such as anaggregator, that is in charge of the content control and/or contentsettings for a device environment. For example, a user may set up aparental control at one device, and the one device then becomes theinstigator of content control for other media playback devices of theenvironment. Also, where digital rights are required for playback, theinstigator of content control may manage those digital rights to preventunauthorized playback.

The logical operations begin at content operation 1102 where a firstdevice is attempting to play media. Here, the first device obtainscontent information included within the media, such as recognizing thetitle of a CD or DVD that is about to be played. Obtaining the contentinformation applies for devices that support multiple media formats,such as a DVD player obtaining content information from DVDs or audioCDs during playback. Then, at query operation 1104, the first devicedetects whether it has its own content rules. If so, then the firstdevice detects whether the content is playable by comparing the contentinformation to the associated content rules. At least two checks may bedone at this point, one for content ratings and one for content rights.Content ratings are limits on the ratings of media that can be played,such as no content worse than a PG rated movie or no content with aparticular type such as excessive adult language. Content rights aredigital rights for playback authorization that prevent copyright orlicense infringement.

If the content is not playable, then the first device stops playback atstop operation 1112. If the content is playable, then two options mayoccur depending upon whether the first device is configured to obeycontent rules from a device environment in addition to its own contentrules. For example, the first device for media playback may be portableand may easily be taken to other device environments that impose morestringent restrictions on media content than the first device imposes onitself. At query operation 1107, the first devices detects whether it isconfigured to obey content rules of the environment in addition to itsown content rules. If the first device is configured to obey only itsown content rules, then the first device begins normal playback of themedia at playback operation 1108. The first device may reference contentrules at this point at settings operation 1110 to determine whether thecontent being played back has an associated preferred setting or settinglimitation. For example, a user may have configured a rule that aparticular movie is to be played back at a preferred volume setting orthat the volume for playback cannot exceed a particular setting. Thefirst device implements the preferred setting or limitation duringplayback.

If the first device is configured to obey its own content rules as wellas the content rules of any environment where it is placed, then afterdetermining that the media content is playable according to its ownrules, operational flow transitions from query operation 1107 to sendoperation 1114. Additionally, if query operation 1104 detects that thefirst device does not have an applicable content rule, then operationalflow transitions directly to send operation 1114.

At send operation 1114, the first device transmits a message having thecontent information previously obtained to a second device thatmaintains content rules for the current environment where the firstdevice is located. The second device receives the message with thecontent information and compares the content information to the storedcontent rules at rule operation 1116. The comparison to the contentrules again involves content ratings and/or rights, settings, and/orsetting limitations. The details of this comparison are also shown inFIG. 11.

The comparison begins at query operation 1126 where the second devicedetects whether the content is playable in relation to the contentrules. The content rules may specify a maximum rating and/or whetherdigital rights exist for the content being played. Other limitations mayalso be specified in the content rules for comparison to the contentinformation, such as a limitation on adult language present in thecontent that is indicated by the content information. If the content isnot playable, then the comparison indicates that a stop instructionshould be sent at stop operation 1130. If the content is playable, thenthe comparison indicates that a play instruction should be sent alongwith any associated settings or setting limitations at playbackoperation 1128.

Once the comparison is complete, the second device directs a message tothe first device at send operation 1118, and the message instructs thefirst device according to the stop instruction or playback instructionresulting from the previous comparison to the content rules. The firstdevice receives the message and interprets the instruction at receiveoperation 1120. The first device then implements the receivedinstruction to either stop playing the media content or begin playbackwith any specified settings or limitations at implementation operation1122.

As an option, the first device may then create a content rule thatassociates the instruction with the content information at ruleoperation 1124 if the first device does not already have a local contentrule. By creating the rule at the first device, the first device will atquery operation 1104 detect that a content rule exists on subsequentattempts to play the same content. The first device will then handle itsown content control without requiring communication with the seconddevice.

The second device may obtain content rules through various methods. Forexample, the second device may receive a message from a third device atreceive operation 1132, and the message specifies a content rule. A usermay have selected a content rule at the third device for media playback,and the third device then provides the rule to the second device as anautomatic function or in response to a request for content rules fromthe second device. The second device creates the content rule by storingit in memory at rule operation 1134.

During playback, the first device periodically repeats the comparison toenvironmental content rules at rule operation 1125, as was initiallydone at rule operation 1116. This operation 1126 is done periodicallybecause if the first device is portable it may change locations afterthe start of playback. In that case, if the playback was initiallypermissible but later becomes impermissible because the first deviceenters a more restrictive device environment, then playback stops asindicated at stop operation 1130.

FIG. 12 shows logical operations demonstrating the borrowing of mediarights of a content rule from a device, such as an aggregator, thatmaintains the content rules. The logical operations begin by a firstdevice sending a request to borrow media rights to a second device thatmaintains the media rights at send operation 1202. For example, thefirst device may be an MP3 player and the request is for permission toplay a particular song or volume of songs.

The second device receives the request and determines if the mediarights to the content exist at receive operation 1204. If so, and theyare not flagged as borrowed, then the second device sends the mediarights to the first device to allow the first device to play the contentat send operation 1206. The media rights are then flagged as borrowed atthe second device at flag operation 1208. Subsequently, when a thirddevice requests authorization for media playback from the second devicefor the same content at send operation 1210, the second device thenchecks the media rights at test operation 1212 and detects the flag. Thesecond device then sends a stop instruction to the third device at sendoperation 1214 to prevent the third device from playing the contentbecause the first device already has rights to it.

These logical operations could also be adapted to provide a count forthe media rights so that more than one device can access the mediarights for playback of content if multiple rights are owned for thecontent. Each time the media rights are borrowed by a device, the countof media rights is decremented. Once the count reaches zero, stopinstructions are sent to the devices subsequently attempting playback.Furthermore, there can be a similar device exchange to unflag thedigital rights or increment the count to restore capability of otherdevices to subsequently borrow the digital rights to media content.

The device interactions discussed above including general interaction tobring about states changes, interactive learning of interaction rules,and interactive content control become increasingly complicated as thenumber of devices in the environment increase. As shown in FIG. 13, whenthe number of devices of an environment 1300 grows to six, the number ofbi-directional communication paths grows to fifteen to ensure that everydevice can communicate directly with every other device. Each deviceuses five bi-directional paths (a first device 1302 uses paths 1314,1316, 1318, 1320, and 1322; a second device 1304 uses paths 1314, 1324,1326, 1328, and 1330; a third device 1306 uses paths 1316, 1324, 1332,1334, and 1336; a fourth device 1308 uses paths 1318, 1326, 1332, 1338,and 1340; a fifth device 1310 uses paths 1320, 1328, 1334, 1338, and1342; and a sixth device 1312 uses paths 1322, 1330, 1336, 1340, and1342).

The complexity in coordinating the communications and interactions insuch a crowded environment 1300 may result in inadequate bandwidth forthe communication channels, cross-talk between the channels, andincompatible transports between devices. Furthermore, unintended rulesmay be learned because one or more of the devices may be unrelated tothe others. For example, one person may answer the telephone shortlybefore another person starts the clothing dryer. There was no intendedrelationship but the phone or the dryer may associate the two statechanges as an interaction rule, which the users never intended.

An aggregator 1402 as shown in FIG. 14 may be introduced into a crowdedenvironment 1400 to alleviate one or more of the concerns. As shown inFIG. 14, an aggregator 1402 can be used to reduce the number ofbi-directional communications paths. For the six device environment, theaggregator has reduced the number of paths down to six (device 1414 usespath 1426, device 1416 uses path 1428, device 1418 uses path 1430,device 1420 uses path 1432, device 1422 uses path 1434, and device 1424uses path 1436). The aggregator 1402 acts as a conduit of communicationfrom one device to another, and may also be configured to control orotherwise manage functions of a single device.

The aggregator 1402 uses a transmitter 1408 and receiver 1406 capable ofcommunicating with the multiple devices. The transmitter 1408 andreceiver 1406 may be configured to receive from all devices usingvarious techniques known in the art. For example, frequency divisionmultiplexing, time division multiplexing, code division multiplexing,optical multiplexing, and other multiplexing techniques may be used fora particular environment 1400 so that multiple devices can communicatewith the aggregator 1402.

The aggregator 1402 also has a processor 1404 and a memory 1410. Theprocessor 1404 communicates with the transmitter 1408, receiver 1406,and memory 1410 through a bus 1412. The aggregator 1402 may beincorporated into a particular device of the environment as well so thatthe aggregator includes the additional device features 4such ascomponents and a state sensor discussed in relation to FIG. 2. Thelogical operations of an aggregator such as shown in FIG. 14 arediscussed below.

The processor 1404 of the aggregator may be configured to performvarious advanced functions for the device environment. The processor1404 may be configured to perform periodic review of interaction rulesto edit rules that are inappropriate for various reasons. For example,memory 1410 may contain a list of impermissible associations that theprocessor 1404 may refer to when reviewing interaction rules. If animpermissible association is found the, communication link that causesthe problem may be excised from the rule. Additionally, the processor1404 may be configured to entirely remove interaction rules that areinappropriate.

The processor 1404 may also be configured to support complex interactionrules. For example, devices may be separated into classes so thatactions of one device may only affect devices within the same class. Theprocessor 1404 may reference such class rules in memory 1410 to filterout fauly rules that might otherwise be learned, such as those wheredevices of different classes are involved. Furthermore, the processor1404 may be configured to develop rules based on conditional logic orinterative logic, and perform multiple activities of a rule in series orin parallel.

As an example of conditional logic being employed, a rule may specifythat a phone ringing means the volume should be decreased for severaldifferent devices but only if they are actively playing content. Thenwhen the phone hangs up, the volume should be increased but only forthose devices whose volume was decreased by the phone ringing. Anexample of interative logic provides that the front porch lights shouldbe turned on at 6 p.m. and off at 12 a.m. everyday.

An example of serial execution of interaction rules with multipleactivities provides that when a light on a computer desk is turned on,the computer is then turned on, and after that the monitor is turned onfollowed by the computer speakers being turned on. An example ofparallel execution of interaction rules with multiple activitiesprovides that when a person is exiting a room, all devices of the roomare powered off simultaneously.

FIG. 15 illustrates an embodiment of an aggregator 1502 that isadditionally configured to translate among different communicationtransports of the device environment 1500. Device 1518 may communicatethrough signals 1524 of a first communication transport while device1522 communicates through signals 1522 of a second communicationtransport. For example, the first communication transport may beEthernet while the second communication transport is fiber optical. Thecommunication transports may differ in the physical mode of transferringsignals (e.g., Ethernet versus fiber optical) and/or in the logical mode(a first data encoding scheme versus a second).

The aggregator 1502 includes a first transmitter 1508 and receiver 1506,separate or combined as a transceiver, for communicating across thefirst communication transport. The aggregator may also include a secondtransmitter 1512 and receiver 1510, separate or combined as atransceiver, for communicating across the second communication transportwhere the second communication transport differs in the physical mode oftransport. A processor 1504 communicates with memory 1514 and the twotransmitter-receiver pairs through a bus 1516. Although twotransmitter-receiver pairs are shown for two communication transports,one skilled in the art will recognize that any number oftransmitter-receiver pairs and communication transports may be utilized,including only one, depending upon the different number of physicaltransports to support within the device environment.

The processor 1504 detects from the messages being received wherecommunications should be directed. This includes determining whether themessages should be translated to a new communication transport whensending the message to the intended device. The processor 1504 mayperform the same logical operations of the processor of aggregator 1402with the addition of translation operations from one transport toanother where necessary.

FIG. 16 shows the basic logical operations of an aggregator. The logicaloperations begin when a first device transmits a message to theaggregator at send operation 1602. The first device may send a messageto the aggregator that is intended as a broadcast to all devices, as amessage directed to a specific device, or as a message intended solelyfor the aggregator. The aggregator receives the message at receiveoperation 1604.

The aggregator then references interaction rules that it maintains inmemory in relation to the message it has received at rule operation1606. For example, the environment may be configured so that the devicesmaintain no interaction rules other than to direct a message for everystate change to the aggregator and rely solely on the interaction rulesof the aggregator to bring about subsequent activity in the environment.The environment may alternatively be configured where the devicesmaintain interaction rules and provide instruction to the aggregatorwith each message, so that the aggregator acts upon the instruction tobring about subsequent activity.

After the aggregator has received the message and referred to theinteraction rules in relation to the message, the aggregatorcommunicates with devices of the environment in accordance with theinteraction rules and any received instruction from the first device atcommunication operation 1608. For example, the aggregator may possessthe interaction rule that when the VCR is on, the TV should be tuned tochannel 3. When the aggregator receives a power on message from the VCR,the aggregator then sends an instruction to the TV to tune to channel 3.Alternatively, the power on message may instruct the aggregator to sendan instruction to the TV to tune in channel 3.

FIG. 17 shows the logical operations of an embodiment of an aggregator,such as the aggregator 1502 of FIG. 15. The logical operations begin atdetect operation 1702 where the first device detects its own statechange. The first device then sends a message to the aggregator at sendoperation 1704. The aggregator receives the message at receiveroperation 1706 and references its interaction rules at rule operation1708 in relation to the received message indicating the state change.

The aggregator tests whether to switch communication transports at queryoperation 1710 by referencing its interaction rules. The interactionrules specify how to communicate with each device. The aggregator learnsthe one or more devices to communicate with in response to the messagefrom the first device by either looking up the state change of the firstdevice in the interaction rules to find associations or by interpretingan instruction from the first device included in the message. Afterdetermining the proper device to communicate with, the aggregator canlook up the device in memory to determine which communication transportto employ.

Once the aggregator has determined which transport to use for the nextcommunication, the message from the first device or a new message fromthe aggregator is prepared by translating to the second communicationtransport appropriate for the next communication at translate operation1712. Where only the logical mode of communication transport differs, asecond communication transport may not be needed. Furthermore, theaggregator may act as a conduit where no change in the physical orlogical mode of transport should occur. As an example of where a changein transport does occur, the aggregator may receive a message from theVCR via infrared airwave signals and then prepare a message to the TV tobe sent via a fiber optical connection. The aggregator sends the messageto the second device at send operation 1714. The second message mayinstruct the second device that the first device has changed state ifthe second device has its own interaction rules, or the message mayprovide a specific instruction to the second device.

After receiving the message, the second device implements anyinstruction or automatic state change dictated by its own interactionrules. The second device may respond to the aggregator if necessary atsend operation 1716. The return message may be an indication to theaggregator of the state change that the second device has performed ormay be a reply to a request from the aggregator such as for currentstate, capabilities, or rules. The aggregator again references itsinteraction rules at rule operation 1718 to determine the next actionafter receiving the message from the second device. The aggregator thencommunicates with other devices of the environment as necessary atcommunicate operation 1720.

The logical operations for the aggregator learning the interaction rulesbeing applied are also shown in FIG. 17. Several possibilities exist forlearning rules at the aggregator. A user interface discussed below maybe provided so that a user enters interaction rules at user operation1722. The aggregator may observe closely occurring state changebroadcasts that are associated as interaction rules at observationoperation 1724, as was discussed above for learning with individualdevices. The aggregator may request that a particular device forward itsinteraction rules to the aggregator where they can be stored andimplemented at request operation 1726.

After receiving the interaction rule in one of the various ways, theaggregator stores the interaction rule at rule operation 1728. Whenstate change messages are received at the aggregator and the aggregatorreferences the interaction rules such as at rule operation 1708, theaggregator compares information in the message to the stored rules atcomparison operation 1730. Through the comparison, the aggregatordetermines the appropriate action to take to complete communication toother devices.

Device interaction within the device environment allows the burden onthe user to be lessened while the user is present within or absent fromthe environment. However, under certain scenarios the user is absent butneeds to remain in contact with the device environment. For example, theuser may need to know when the oven is finished cooking so the user canreturn home, or the user may need to delay the oven from automaticallypreheating at a certain time because the user will be late. Therefore,for these scenarios the device environment needs to communicate remotelywith the user.

FIG. 18 shows one illustrative case of device communication where themessages extend beyond a closely defined area, such as a single room orhousehold, to an external or broader area. The external areua includesany destination reachable via a communications network. Thus, in thisillustrative case, the device environment is not defined by proximitybut by explicit definition by the user. Such explicit definition may beprovided by the user in many ways, such as through a listing stored inmemory that describes the devices and their address where they may beaccessed through communication networks including the Internet, wirelesscommunication network, and landline telephone network. Thus, as usedherein, device environment should be understood to include bothenvironments defined by proximity as well as explicitly definedenvironments.

Additionally, FIG. 18 shows an illustrative case of device communicationwhere notification messages are passed between a notification devicethat is interfaced with the user and devices of the environment nototherwise interfaced with the same user. Thus, messages may be passed tothe user from devices of the environment and from the user to thedevices without the user interacting directly with those devices thatsend or receive the message. Such notification devices may be external,as discussed above, in that they are not part of the device environmentthrough proximity but by explicit definition by the user, or thenotification devices may be in close proximity and be included in thedevice environment on that basis.

A device 1802 such as an aggregator for sending notifications to thenotification device of the user and/or for communicating to both devicesdefined by proximity and external devices is present in the environment1800. The device 1802 includes at least one transmitter 1814 andreceiver 1812 for communicating with proximity based devices 1818 in theenvironment 1800 over a communication transport 1820. The device 1802 ofincludes a memory 1806 that stores interaction rules and a processor1804 for executing the functions of the device 1802. The processor 1804communicates through the bus 1816. The memory 1806 may also storetranslation rules in the embodiment where communication withnotification devices is supported.

The device 1802 of this embodiment also includes at least onetransmitter 1810 and receiver 1808 that communicate through a remotecommunications transport 1828 to external devices. The remotecommunications transport 1828 may take various forms such as aconventional telephone network 1822 including a central office 1826. Theremote communications medium may additionally or alternatively involve awireless network 1824 for mobile telephones or for pagers.

Communication can be established between the device 1802 and a remotelylocated telephone 1830, computer 1832, or wireless communication device1834 such as a mobile phone or pager which is explicity defined inmemory 1806 as being part of the device environment. The device 1802 canrelay information between itself or other proximity based devices of theenvironment 1800 and the remotely located communication devices.

In the embodiment where notification devices are supported, the user canremain in contact with the device environment 1800 by communicatingthrough the notification devices that are either external, such asdevices 1830–1834, or are proximity based, such as device 1836. Forexample, the device 1802 may send short messages to a mobile phone 1834or to a proximity based portable communication device 1836 if the useris in proximity. The device 1802 may provide machine speech or text thatcan be interpreted by the user as a notification of a state of theenvironment. Similarly, the user may send machine tones, speech, or textback to the device 1802 that can be interpreted by the device 1802 as aninstruction for the environment.

For example, to implement the notification process the processor 1804may recognize a set of voice commands, tones, or text and translatethose into instructions for various devices by referencing translationrules to interpret the instruction. The processor 1804 may thenreference interaction rules to communicate the instruction to theappropriate device based on identification received in the message fromthe remote device. Likewise, the processor 1804 may choose from a set ofmachine voice commands, tones, or text to communicate messages from theenvironment back to the user when the interaction rules indicate thatthe remote device should be contacted.

FIG. 19 shows the logical operations for communication from the deviceenvironment to the notification device 1830–1834 or 1836. The logicaloperations begin at detect operation 1902 where a first device of theenvironment detects its own state change. The first device itself or adedicated device for notification communications such as an aggregatormay then reference rules for interaction to determine whether anotification communication is necessary based on the state change atrule operation 1904. For example, if the previously discussed contentcontrol device detects that unacceptable content playback is beingattempted, a notification may be provided to the notification device1836 or 1830–1834.

The interaction rules may provide a hierarchy of communication withnotification devices, or for other non-notification devices as well, sothat a particular state change may require that communications cyclethrough a list of devices until a response is received or the list isexhausted. At detect operation 1914, the appropriate device of theenvironment determines from the interaction rules the order ofcommunication that should occur. For example, a particular state changemay require that a page be left with the user followed by a call to amobile phone if there is no response to the page within a certain amountof time.

The logical operations of FIG. 19 assume that the notification device isan external device that is explicitly defined by the user. Thus, afterdetermining the one or more notification devices to contact, the deviceof the environment references translation rules at rule operation 1906to determine how to convey the message to the remotely locatednotification device that should be contacted. The translation rules aretypically specified by the user directly at input operation 1916.Through a user interface, the user can specify the hierarchy and theparticular translation rules to use. For example, the user can specifythat a pager is contacted by dialing a specific telephone number overthe ordinary telephone network, and that a text message should be leftupon an answer. Rules may also include constraints such as the range oftime when a particular notification device should be contacted.

The device of the environment executes the interaction rule andtranslation rule to communicate remotely to a second device (i.e., anotification device) at communication operation 1908. As one exemplaryoption where a hierarchy is employed, the device tests whether thesecond device has responded at query operation 1910. If so, then thelogical operations return to await the next state change requiringremote communications. If not, then the device of the environmentcommunicates remotely to a third device (i.e., a different notificationdevice) as specified in the hierarchy at communication operation 1912again with reference to the interaction and translation rules. Cyclingthrough the devices of the hierarchy continues until query operation1910 detects a response or the list is exhausted.

Another exemplary option is to communicate with one or more notificationdevices without regard to a hierarchy. After the device of theenvironment has provided a communication to the second notificationdevice, then a communication is automatically provided to the thirdnotification device at communication operation 1912. This continues foras many remote devices as specified in the interaction rules.

FIG. 20 shows the logical operations for communications from thenotification device back to the device environment. At send operation2002, the notification device directs a message to a first device of theenvironment that completes notification communications. For example,where the notification device is external to the proximity defineddevice environment, the first device may maintain a connection to atelephone line, and the user dials the number for the line to contactthe first device. The first device answers the call and awaits datasignals from the remote notification device. The remote notificationdevice then provides the message by the user speaking or using dialingtones.

The first device receives the message at receive operation 2004 andtranslates the message for transport to a second device of theenvironment at translate operation 2006. The first device may translatethe message by referencing translation rules to convert the message intoa form usable by the second device and by referencing interaction rulesto determine that the second device should be contacted. For example,according to the translation rules, an initial “1” tone from the remotedevice may indicate that the oven should be contacted, and a subsequent“2” tone from the remote device may indicate that the oven should cancelany automatic preheating for the day.

Thus, translate operation 2006 involves determining the second device tocommunicate with through detecting an ID of the second device from themessage of the remote device at ID operation 2020. In the example above,the ID of the oven is an initial “1” tone. The first device receives the“1” tone and references a “1” tone in the interaction rules to determinethat a message should be sent to the oven. The first device receives the“2” tone and, knowing that the message is for the oven, references a “2”tone for the oven in the translation rules to determine that a cancelpreheat message to the oven is necessary. The message is communicatedfrom the first device to the second device at communication operation2008.

The second device receives the message and implements the instruction atimplementation operation 2010. For the example above, the oven receivesa message instructing it to cancel its pre-programmed preheat operationfor the day, and it cancels the preheat operation accordingly. As anexemplary option to the logical operations, the second device may thensend a message back to the first device confirming it has implementedthe instruction at send operation 2012.

The first device 2014 receives the message from the second device atreceive operation 2014, and then the first device 2014 translates theconfirmation to a message that can be sent to the notification device attranslate operation 2016 in the instance where the notification deviceis external. For example, the first device 2014 may determine from thetranslation rules that it should send a pattern of tones to thetelephone used to place the call to signal to the user that the ovencanceled the preheat operation. The first device 2014 then communicatesthe message to the remote notification device over the remotecommunication transport at communication operation 2018 to complete thenotification communications.

The user may be provided a user interface to interact directly withdevices of the environment such as the aggregator. As discussed above,the user may program interaction rules, translation rules, andhierarchies for remote communication through the user interface.Additionally, the user may review information about the deviceenvironment through the user interface, such as current states ofdevices and existing interaction rules and translation rules of theenvironment.

FIG. 21 shows the major components of an exemplary device 2102establishing a user interface for the device environment. The userinterface 2102 may be a separate device or may be incorporated into adevice of the environment such as an aggregator. The user interface 2102includes a processor 2104 for implementing logical operations of theuser interface. The processor 2104 communicates with a memory 2106 and adisplay adapter 2108 through a bus 2110. The processor 2104 referencesthe rules stored for the environment in the memory 2106 to provideinformation to the user on a display screen 2112 driven by the displayadapter 2108.

The user interface 2102 may provide several mechanisms for receivinguser input. As shown, a touchscreen 2112 is provided so that the usercan make selections and enter information by touching the screen 2112that displays selectable items such as text or icons. One skilled in theart will recognize that other user input devices are equally suitable,such as but not limited to a keyboard and mouse.

Several exemplary screenshots of the user interface are shown in FIGS.22–25. The screenshots demonstrate a graphical user interface that isicon based. However, other forms of a user interface on screen 2112 arealso suitable, such as a text-based user interface. Furthermore, manyvariations on the graphical user interface shown are possible.

FIG. 22 shows a screenshot 2200 that contains icons that formrepresentations of the devices present within the environment. As shown,six devices are present within the environment and the screenshot 2200includes a television representation 2202, a VCR representation 2204, amicrowave representation 2206, a stove/oven representation 2208, awasher representation 2210, and a dryer representation 2212. Alsoincluded in the screenshot 2200 are a rule button 2214, a first learnmode button 2216, and a second learn mode button 2218.

From screenshot 2200, the user may make a selection of a devicerepresentation to learn information about the device such as its currentstate. The logical operations of viewing device information arediscussed in FIG. 27. The selection may also be used to send aninstruction to the device to immediately bring about a state change asmay be done with an ordinary remote control. The user may select therule button 2214 to view interaction or translation rules already storedand being executed for the environment. An example of viewing existingrules is discussed in more detail with reference to FIG. 25.

The user may also make a selection of the first learn mode button 2216to program an interaction or translation rule by interacting with devicerepresentations and function representations for the device. The firstlearn mode is discussed in more detail with reference to FIG. 23 and thelogical operations of FIG. 28. Additionally, the user may make aselection of the second learn mode button 2218 to program an interactionrule by interacting with the device itself. The second learn mode isdiscussed in more detail with reference to FIG. 24 and the logicaloperations of FIG. 26.

FIG. 23 shows a screenshot 2300 after a user has selected the TVrepresentation 2202 from the initial screenshot 2200. The screenshot2300 shows the device representation or icon 2302 and the associatedfunction representations or icons for the functions of the TV present inthe environment. The function representations include channel selectionrepresentation 2304, volume selection representation 2306, muterepresentation 2308, power representation 2312, and signal inputrepresentation 2310. The user makes a selection of a particular functionrepresentation to be associated in an interaction rule and then selectsanother function representation of the TV or another device to completethe rule.

As described above, the user may select a power on representation forthe VCR and then select the channel selection representation 2304 toindicate a channel 3 for the TV. The interaction rule is created as aresult so that whenever the VCR is powered on, the TV automaticallytunes to channel 3. The interaction rule may be programmed to includeadditional associations as well, such as setting the TV volumerepresentation 2306 to a particular volume setting as well once the VCRis powered on. Likewise, rules may be specified for a single device,such as for example specifying that when the TV is turned on, the volumeof the TV should automatically be set to a particular level.

The logical operations for the first learn mode are shown in FIG. 28.The logical operations begin by the user interface displaying the devicerepresentations at display operation 2802. A user selection of a firstdevice selection is selected at input operation 2804. The functionrepresentations of the first device are displayed on the screen for thefirst device at display operation 2806. A user selection of a functionrepresentation for the first device is received at input operation 2808.

The device selections are redisplayed and the user selects a seconddevice representation at input operation 2810. The functionrepresentations of the second device are displayed at display operation2812. A user selection of a second device selection for the seconddevice is received at input operation 2814. The function representationselected for the first device is associated with the functionrepresentation for the second device to create the interaction rule atrule operation 2816.

FIG. 24 shows a screenshot 2400 that is displayed after a user selectsthe second learn mode button 2218. The screenshot 2400 includes a button2402 that is a choice to learn a first portion of the interaction rule.The user presses the button 2402 and then selects the first function onthe first device itself within the environment. In response to the firstdevice providing a message about its resulting state change, theselected function is displayed in field 2404.

The user then presses the button 2406 that is a choice to learn a secondportion of the interaction rule. The user selects the second function onthe second device itself, and in response to the second device providinga message about its state change, the selected function is displayed infield 2408. The interaction rule is created by associating the functionshown in the first display field 2404 with the function shown in thesecond display field 2408. In the example shown, the rule that resultsis if the VCR is powered on (a first received state change message),then the TV tunes to channel 3 (a second received state change message).

The development of the rule may continue as well. The user may press thebutton 2410 that is a choice to learn a third portion of the interactionrule. The user selects the third function on the third device itself,and in response to the third device providing a message about its statechange, the selected function is displayed in filed 2412. In the exampleshown, the rule that results is if the VCR is powered on, then the TVtunes to channel 3 and then the VCR begins to play. Additionally, asdiscussed above in relation to advanced interaction rules of theaggregator, the user may specify via buttons 2414, 2416 whether theexecution of the multiple step interaction rule should be performed inparallel or serial fashion. If in parallel, then turning the VCR oncauses messages to be simultaneously instructing the TV to tune tochannel 3 and the VCR to begin playing simultaneously. If in series,then the TV is instructed to turn on prior to the VCR being instructedto play.

The logical operations of an example of the second learn mode are shownin FIG. 26. The logical operations begin at choice operation 2602 wherethe first choice is provided to the user for selection to initiatelearning of the first portion of the interaction rule. The user selectsthe first choice, which is received at input operation 2604. The userthen selects the first function on the first device itself at inputoperation 2606. A state change message from the first device is receivedat the device creating the rule at receive operation 2608, and themessage indicates the function the user selected. The functiondescription is stored in memory.

The second choice is provided to the user for selection to initiatelearning of the second portion of the interaction rule at choiceoperation 2610. The user then selects the choice at input operation 2612to initiate learning of the second portion of the interaction rule. Theuser then selects the second function representation on the seconddevice at input operation 2614. A state change message is received fromthe second device at the device creating the rule at receive operation2616, and the message indicates the function the user selected. Thefunction description is stored in memory. Once the two functiondescriptions are known by the device creating the rule, the firstfunction description is associated with the second function descriptionat rule operation 2618 to create the reaction rule.

FIG. 25 shows a screenshot 2500 that results from the user selecting therule button 2214 and a selection of a device that is involved in therule. For example, the user may select the TV representation 2202 toview an interaction rule involving the TV present in the deviceenvironment. A TV representation 2502 is displayed and is connected to afunction representation 2504 that indicates that the TV is being tunedto channel 3. A VCR representation 2506 is displayed and is connected toa function representation 2508 that indicates that the VCR is beingpowered on.

A connector 2510 is shown connecting the VCR function representation2508 to the TV function representation 2504. As shown, the connector2510 is directional as an arrowhead points to the TV functionrepresentation 2504 to indicate that the TV function results from theVCR function. The corresponding interaction rule provides theassociation of VCR on to TV channel 3 only to automatically control theTV in response to the VCR but not the other way around. Thus, when theTV is tuned to channel 3 by the user, the VCR does not automaticallyturn on because the interaction rule is learned as a directionalassociation.

Other interaction rules may involve a connector that is not directionalso that the association is absolute rather than directional. Forexample, it may be preferred that the VCR automatically turn on when theTV is tuned to channel 3 and that the TV automatically tune to channel 3when the VCR is turned on. Such an interaction rule would be absoluterather than directional, and the connector 2510 would lack an arrowheador alternatively have arrowheads pointing in both directions. Oneskilled in the art will recognize that other visual connectors besidesarrowheads and lines are suitable as well.

To view information about a specific device in the environment, the usermay select the device representation from the screenshot 2200 of FIG.22. A screenshot such as the screenshot 2300 of FIG. 23 will bedisplayed. Along side each function representation, the value for thatfunction may be displayed to inform the user of the current state of thedevice. For example, a 3 may appear next to the channel representation2304 while a checkmark appears next to the mute representation 2308 toindicate that the TV is currently tuned to channel 3 but is muted.

The logical operations for obtaining information about a device throughthe device interface are shown in FIG. 27. The logical operations beginby displaying a choice of device representations at display operation2702. A user selection of a device representation is received at inputoperation 2704 to select a first device. A message is then sent from theuser interface, such as an aggregator, to the first device selected bythe user at send operation 2706. The message includes a request forinformation, such as the current status, from the first device.

The first device receives the request for information at receiveoperation 2708. The first device then directs a reply back to the userinterface at send operation 2710. The reply is a response to the requestfor information and includes the current status information of the firstdevice. The user interface device receives the reply from the firstdevice at receive operation 2712, and then the user interface displaysthe current status information on the screen at display operation 2714.

Various embodiments of devices and logical operations have beendiscussed above in relation to communications between devices, automaticand manual learning of interaction rules, content control, aggregatorfunctions, remote communications, and a user interface. Although theseembodiments of devices and logical operations may be combined into arobust system of device interaction, it should be noted that variousdevices and logical operations described above may exist in conjunctionwith or independently of others.

Although the present invention has been described in connection withvarious exemplary embodiments, those of ordinary skill in the art willunderstand that many modifications can be made thereto within the scopeof the claims that follow. Accordingly, it is not intended that thescope of the invention in any way be limited by the above description,but instead be determined entirely by reference to the claims thatfollow.

1. A method of establishing interaction with a notification deviceinterfaced with a user and an environment having a plurality of deviceswhere each device performs one or more functions within the environmentwith the one or more functions of each device being associated withvarious states, the method comprising: detecting a change of state at afirst device; in response to detecting the change of state, referencingrules of device interaction that govern communication between the firstdevice and a notification device, the rules of device interactionproviding an association between the detected change of state of thefirst device, the notification device that is communicated with inresponse, and the content of the communication to the notificationdevice interfaced with the user, wherein referencing the rules of deviceinteraction further comprises detecting from the rules a hierarchy ofnotification devices and non-notification devices to communicate with inresponse to the detected change of state; and communicating with thenotification device according to the rules of device interaction inresponse to detecting the change of state.
 2. The method of claim 1,wherein the notification device is located remotely, and wherein themethod further comprises referencing rules of translation that indicateinstructions for conveying information remotely to the notificationdevice through a remote communications transport.
 3. The method of claim1, further comprising: detecting whether a first notification device isresponsive to the communication; and when the first notification deviceis not responsive, communicating with a second notification deviceaccording to the rules of device interaction.
 4. The method of claim 1,further comprising communicating with a first notification device andcontemporaneously communicating with a second notification device. 5.The method of claim 2, further comprising receiving user input to createthe translation rule for the second device.
 6. The method of claim 2,wherein information is conveyed remotely through email.
 7. The method ofclaim 2, wherein information is conveyed remotely through electronicpages.
 8. A device for establishing interaction between a notificationdevice interfaced with a user and a plurality of devices within anenvironment where each device performs one or more functions within theenvironment with the one or more functions of each device beingassociated with various states, the device comprising: a firsttransmitter for a communication transport of the environment; a firstreceiver for the communication transport of the environment; a memorythat stores interaction rules and translation rules for communicatingbetween the notification device and the plurality of devices of theenvironment; and a processor in communication with the firsttransmitter, the first receiver, and memory, the processor beingconfigured and to direct messages between devices of the environment andthe notification device according to the interaction rules andtranslation rules, wherein the interaction rules specify when acommunication to a notification device should occur and the translationrules specify the content to include in the communication to thenotification device, wherein the processor is further configured todetect from the interaction rules a hierarchy of notification devicesand non-notification devices to communicate with in response to adetected change of state of a device of the plurality.
 9. The device ofclaim 8, wherein the notification device is remotely located andcommunicates through a remote communications transport, the devicefurther comprising: a second transmitter for the remote communicationtransport; a second receiver for the remote communication transport; andwherein the processor is further configured to translate messagesbetween the communication transport of the environment and the remotecommunication transport according to the translation rules.
 10. Thedevice of claim 8, wherein the processor is further configured to send amessage to a first notification device of the hierarchy, detect whetherthe first notification device is responsive to the communication, andwhen the first notification device is not responsive, communicate with asecond notification device of the hierarchy according to the rules ofdevice interaction and the rules of translation.
 11. The device of claim8, wherein the processor is further configured to receive user input tocreate the translation rules and to store the translation rules createdfrom the user input in the memory.
 12. The device of claim 9, whereinthe remote communication transport is email system.
 13. The device ofclaim 9, wherein the remote communication transport is an electronicpaging system.
 14. The device of claim 9, wherein the remotecommunication transport is the Internet.
 15. A computer-readable mediumhaving stored thereon computer-executable instructions which, whenexecuted by a computer, cause the computer to: detect a change of stateat a first device; in response to detecting the change of state, toreference rules of device interaction that govern communication betweenthe first device and a notification device, the rules of deviceinteraction providing an association between the detected change ofstate of the first device, the notification device that is communicatedwith in response, and the content of the communication to thenotification device interfaced with the user, wherein referencing therules of device interaction further comprises detecting from the rules ahierarchy of notification devices and non-notification devices tocommunicate with in response to the detected chance of state; andcommunicate with the notification device according to the rules ofdevice interaction in response to detecting the change of state.
 16. Thecomputer-readable medium recited in claim 15, wherein the notificationdevice is located remotely, and further comprising computer-executableinstructions stored thereon which, when executed by a computer, causethe computer to reference rules of translation that indicateinstructions for conveying information remotely to the notificationdevice through a remote communications transport.
 17. Thecomputer-readable medium recited in claim 15, further comprisingcomputer-executable instructions stored thereon which, when executed bya computer, cause the computer to: detect whether a first notificationdevice is responsive to the communication; and when the firstnotification device is not responsive, to communicate with a secondnotification device according to the rules of device interaction. 18.The computer-readable medium recited in claim 15, further comprisingcomputer-executable instructions stored thereon which, when executed bya computer, cause the computer to communicate with a first notificationdevice and to contemporaneously communicate with a second notificationdevice.
 19. The computer-readable medium recited in claim 16, furthercomprising computer-executable instructions stored thereon which, whenexecuted by a computer, cause the computer to receive user input tocreate the translation rule for the second device.
 20. Thecomputer-readable medium recited in claim 16, wherein information isconveyed remotely through email.
 21. The computer-readable mediumrecited in claim 16, wherein information is conveyed remotely throughelectronic pages.
 22. The method of claim 1, wherein, for the detectedchange of state, the referenced rules of device interaction require thata page be left with a user followed by a call to a mobile phone if thereis no response to the page within a certain amount of time.
 23. Thecomputer-readable medium recited in claim 15, wherein, for the detectedchange of state, the referenced rules of device interaction require thata page be left with a user followed by a call to a mobile phone if thereis no response to the page within a certain amount of time.